Papers with Large pretrained models
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |
Hierarchical Recurrent Aggregative Generation for Few-Shot NLG (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches do not account for the fact that some sub-tasks, specifically aggregation and lexicalisation, can benefit from transfer learning in different extents. |
| Approach: | They propose a hierarchical approach for few-shot and zero-shot generation using a three-moduled jointly trained architecture. |
| Outcome: | The proposed approach achieves state-of-the-art on few-shot and zero-shot settings compared to previous approaches. |